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ABSTRACT 

Projection  pursuit  algorithms  approximate  a  function  of  p  variables 
by  a  sum  of  non-linear  functions  of  linear  combinations 


(1) 


\ 


11 

f  (x, ,  . . . ,  x  )  -  7  g.  (a . , x,  +  . . .  +  a .  x  )  .  j 

1  p  ®i  il  1  ip  p 


We  develop  some  approximation  theory,  give  a  necessary  and  sufficient  condi¬ 
tion  for  equality  in  (1),  and  discuss  non-uniqueness  of  the  reo resen tat ion. 
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J •  Introduction  and  Statement  of  Main  Results 

We  present  some  mathematical  analysis  for  a  class  of  curve  fitting 
algorithms  labeled  "projection  pursuit"  algorithms  by  Friedman  and  Stuetzle 
(1981  a,  b) .  These  algorithms  approximate  a  general  function  of  p  variables 
by  a  sum  of  non-linear  functions  of  projections: 


(1.1) 


f(x 


1’ 


x  ) 


l 

i-1 


8i<ailxl  +  -*-+aipV 


In  (1.1),  f  is  a  given  function  and  univariate,  non-linear  functions  g^ 

and  linear  combinations  a. ,x, +  . .  .  +  a.  x  are  sought  so  that  a  reasonable 

IX  x  ip  p 

approximation  is  attained.  Such  approximation  is  computationally  feasible 
and  performs  well  in  examples  of  non-linear  regression  with  noisy  data,  high 
dimensional  density  estimation,  and  multidimensional  splines.  In  addition 
to  the  articles  of  Friedman  and  Stuetzle  cited  above  see  Friedman  and  Tukey 
(1974),  Friedman,  Gross  and  Stuetzle  (1983)  for  examples  and  computational 
details.  Huber  (1981  a,  b)  begins  to  connect  the  algorithms  to  statistical 
theory.  This  note  treats  the  algorithms  from  the  point  of  view  of  approxi¬ 
mation  theory. 

It  is  easy  to  show  that  approximation  is  always  possible. 


THEOREM  1.  Functions  of  the  form  Eot^e-  with  real,  a  a  vector  of 
nonnegative  integers,  and  x  »  (x^,  ...,  x^)  are  dense  in  the  continuous  real 
valued  functions  on  [0,  1]P  under  the  maximum  deviation  norm. 


Proof.  The  functions  e~*~  separate  points  of  [0,  1]P  and  are  closed  under 
multiplication.  Finite  linear  combinations  of  such  functions  form  a  point 
separating  algebra  which  is  dense  because  of  the  Stone-Wierstrass  theorem.  ■ 


THEOREM  2.  Functions  of  the  form 

Ea^  cos  (2  it  a*  •  x)  +  0^  sin  (2  ir  b1,  •  *) 
are  dense  in  L^[0, 1]**  . 

Proof.  Any  function  in  L  [0,  l]p  can  be  veil  approximated  by  its  Fourier 
expansion.  See  Volume  2  of  Zygmund  (1959)  and  the  survey  article  by  Ash 
(1976)  for  further  details  and  refinements.  ■ 

Sometimes  equality  is  possible  in  (1.1).  For  example 

1  ,  .  .2  1 .  .2 

xy  -  j  (x+y)  -  ^  (x-y) 

max(x,  y)  -  -j  |x+  y !  +  -  |x  -  y  | 

21  47  41  U  2^  14 

(xy)  “7<x+y)  + - q(*-y) - r(x+2y) - ;(x+ry)  . 

4  •  3  2  •  3J  3 


In  what  follows  we  will  focus  on  conditions  for  equality  in  (1.1)  ae  a  method 
of  determining  examples  to  test,  compare,  and  evaluate  algorithms.  Consider 
first  a  smooth  function  of  2  variables  of  the  special  form 


Clearly 


f(x»  y)  *  g(ax+  by)  . 


If  f  has  the  form 


f(*»  y )  -  1  g. (a.x  +  b.y) 
i-1  1  1 


(1.2) 
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then  the  differential  operator 


L  ■ 


an 

Ci  axV1 


applied  to  f  is  identically  zero.  The  next  theorem  gives  a  converse. 

k  2 

THEOREM  3.  Let  f  €  C  [0,  1]  .  Suppose  that  for  some  real  numbers  cft,  . . . ,  c  , 

On 

n  gk 

the  operator  £  c  : — ■— *7  applied  to  f  is  identically  zero.  If  the  polyno- 

n  ii=0  1  9x1ayn'1 

niial  J  c  z  has  distinct  real  zeros  then  (1.2)  holds  for  some  (a  ,  b  )  .  The 
1=0  11 
lines  a^x  +  b^y  are  all  distinct. 

Theorem  3  is  proved  in  Section  2  which  also  contains  a  discussion  of 
techniques  for  finding  directions  (a^,  b^)  given  f  .  Some  applications  of 
Theorem  3  are  contained  in  the  following  examples. 

APPLICATION  1.  The  functions  e**  and  sin  xy  cannot  be  written  in  the  form 

3n 

(1.1)  for  any  finite  n  .  Indeed,  the  equation  Ec.  — j - r  f  =  0  implies 

3x  3yn 

c±  =  0  and  the  associated  polynomial  has  complex  roots. 

APPLICATION  2.  Let  f(x,y)  be  a  polynomial  of  degree  m.  Then 


y)  -  l  g,(a  x+b.y) 
i-1  1 

where  each  g^  is  a  polynomial  of  degree  at  most  m .  This  follows  by  elimi¬ 
nating  manipulations  from  Theorem  3.  Thus,  any  polynomial  in  2  variables 
can  be  represented  exactly.  Since  polynomials  are  dense  in  C[0,  1]  ,  this 

gives  another  proof  of  denseness  of  projection  pursuit  approximations. 
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APPLICATION  3.  Representations  of  the  fora  (1.1)  are  not  necessarily  unique. 
For  example 

2  2 
xy  “  c(ax  +  by)  -  c(ax-by) 

2  2 

for  any  a  and  b  satisfying  ab  fl  0,  a  +  b  -  1  with  c  -  l/4ab . 

Writing  a  «  cos  0,  b  •  sin  0  ,  any  non-coordinate  direction  can  be  chosen 

for  the  quadratic  g^  .  The  second  direction  is  forced  as  orthogonal  to  this. 

This  suggests  that  substantive  interpretation  of  the  linear  combinations 

(a^ ,  fe4)  is  difficult.  For  a  more  ambitious  example,  consider  the  function 
2 

(xy)  .  This  is  of  4th  degree.  Use  of  Theorem  3  as  outlined  in  Section  2, 

2 

shows  that  (xy)  cannot  be  expressed  as  a  sum  of  n  ■  3  or  fewer  terms  in 
(1.1).  Four  terms  of  4th  degree  suffice: 

(xy)2  -  o1(x+b1y)4  +  a2(x  +  b2y)4  +  a3(x  +  b3y)4  +  a4(x  +  b4y)4  , 

where  b^,  b2>  b^,  b4  are  chosen  as  distinct,  and  satisfying 

blb2  +  blb3  +  blb4  +  b2b3  +  b2b4  +  b3b4  “  0  * 


Then  a^,  a2>  ct^,  a4  are  determined  by 


a 

1 


1 

6 


n*(bj  -  bt) 


where  the  sum  and  product  are  over  J  f  i  .  This  clearly  defines  a  three 
dimensional  family  of  solutions. 
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APPLICATION  4.  Even  if  Che  directions  are  fixed,  Che  representation 

need  not  be  unique.  Suppose  that  n  is  the  smallest  integer  such  that 


If  also 


then 


n 

f(*.  y)  •  I  g.,  (a,x  +  b  .y)  . 
i-1  1  1  1 


n 

f(x.  y)  ■  l  h  (a.x+  b.y)  , 
i-1  1 


ft(t)  -  h±(t:)  -  pt(t)  ,  1  <  i  <  n 


with  a  polynomial  of  degree  at  most  n  -  1  .  The  polynomials  p^  can  be 


chosen  in  an  arbitrary  way  subject  to  the  constraint  Ep^  =  0 .  In  particular, 
any  n  -  1  of  the  p^  can  be  chosen  arbitrarily  and  a  final  polynomial  can  be 
found  to  satisfy  the  constraint.  These  results  all  follow  easily  from  Theorem  3; 


Indeed  the  operator  L  -  II 

1  J*i 


f,  3  3 

l  b  j  3x  aj  3y 


applied  to  f(x,  y)  gives 


i4"~1)(vt+v>  ■  8in"1><V'bi!')J51(Vi'ajbi)- 


The  products  are  non-vanishing  because  the  directions  are  distinct.  It  follows 
that  h^  differs  from  g^  by  at  most  a  polynomial  of  degree  n-1  ,  and  that  an 
arbitrary  polynomial  may  be  added  subject  to  the  constraint. 

In  the  special  case  n  -  2  ,  Theorem  3  was  given  by  Dotson  (1968)  who 
suggests  further  application  to  factoring  probability  densities  and  separation 
of  variables. 

The  generalisation  to  dimension  greater  than  two  is  not  as  neat.  We  give 

3 

the  result  in  3  dimensions,  characterizing  functions  on  [0,  1]  of  the  form 


8i(ail*l+ai2X2+8i3X3)  * 


(1.3) 
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Clearly  a  smooch  function  f(x^,  x^,  x^)  is  of  the  form  g(x^)  if  and 
3  3 

only  if  -5 —  f  and  —  f  vanish  identically.  It  is  equivalent  to  insist  that 
|  b3  -j~  +  g~  +  bj  gj-j  f  vanishes  identically  for  all  (b^,  b^)  in  the 

plane  normal  to  the  x^  axis  (so  b^  ■  0  )  .  The  following  theorem  generalizes 
these  considerations.  The  generalization  to  p-dimensions  is  straightforward. 


3  n  3 

THEOREM  4.  Let  11^  be  n  distinct  planes  in  IRJ  .  Let  fee  [0,  1]  .  Then 

f  has  the  form  (1.3)  if  and  only  if  for  all  b^e  n  ^  , 


(1.4) 


”  { 
i-1  v 


9  9  _9_ 

bil  3xx  +  bi2  3x2  +  bi3  3x3 


|f  H  0  . 


Remarks.  If  ci*di  €  form  a  basis,  (1.4)  holds  for  all  b^  e  11^,  1  <.  i  <  n 
if  and  only  if  it  holds  for  the  2n  cases  in  which  b^  runs  over  possible  basis 
vectors.  The  case  n  -  2  in  (1.3)  is  degenerate  and  may  be  treated  by  Theorem  3 
for  example,  a  necessary  and  sufficient  condition  for 


f(x1,x2,x3)  -  g1(*1)  +  g2(*2)  18  3x“  f  aod  3x7  3x7  f  vanish  identically* 


3_  _3_ 
‘l  3*2 


We  conclude  this  introduction  by  relating  the  above  results  to  Hilbert's 
13th  problem.  In  modern  notation  Hilbert  asked  if  there  are  genuine  multi¬ 
variate  functions.  Of  course,  x  +  y  is  a  function  of  2  variables  but 
xy  »  el°g  x+  logy  a  superposition  of  univariate  functions  and  +. 

Kolmogorov  and  Arnold  showed  that,  in  this  sense,  +  is  the  only  function  of 

2  variables.  They  constructed  S  monotone  functions  [0,  1]  -*■  H,  e  L  ip^  , 

2 

with  the  following  remarkable  property:  for  each  f  e  C[0,  1]  there  is  a 
g  e  C[0,  1]  such  that  for  all  (x,  y) 


f(*»  y) 


♦1(x)  +  \  ^(y)}  • 
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Thus  ^  are  a  "universal  change  of  variables"  which  allows  exact  equality. 

A  nice  discussion  of  this  result  and  its  refinements  can  be  found  in  Lorentz 

(1966,  1980)  and  Vertushkin  (1977).  While  the  functions  and  g  are  given 

In  a  constructive  fashion,  it  does  hot  seem  that  this  result  is  used  to 

approximate  functions  in  an  applied  context.  This  is  probably  because  the 

functions  are  fairly  "wild".  For  example,  it  is  known  that  it  is  not 

possible  to  choose  to  be  functions,  so  fixed  linear  combinations  of 

x  and  y  are  ruled  out.  Indeed,  it  is  known  that  there  is  a  polynomial 

n 

f(x,  y)  for  which  f(x,y)  »  £  g  (a  x  +  b  y)  is  not  possible  with  a  ,  b 

i-1  1  1  1  11 

chosen  independent  of  f  .  In  the  projection  pursuit  approach  to  approximation, 

a^  and  b^  are  allowed  to  depend  on  f  and  Example  2  shows  that  now  any  poly¬ 
nomial  can  be  written  in  the  required  form.  Example  1  shows  that  not  all 
functions  can  be  so  expressed. 

Acknowledgement .  We  thank  Jerry  Friedman,  Bob  Hulqulst,  and  Winnie  Li  for 
helpful  discussions. 


2.  Proof  and  Discussion  of  Theorems  3  and  4.  Let  L  be  the  differential 


operator:  J  c.  — 5 - r  .  By  hypothesis,  the  polynomial 

i-0  1  3x13yn'1 


n  .  .  n 

i  n-i  n 


x^  i 


p  1  n-i  nr  /xi 
1  cxy  -y  l  c  (-J 

i-0  1  i-0  y 


splits  into  distinct  linear  factors.  Thus  L  can  be  written  as 


nl  .  J_  X 

n*  bi  3x  "  ai  3y  J 


,  with  the  lines  a^x  +  b^y  distinct.  It  must  be  shown  that 
n 


f  can  be  represented  as  £  g  (a.x  +  b.y)  .  The  proof  is  by  induction  on  n  . 

i-1  1  1 

For  n  -  1,  suppose  without  real  loss  that  a^  4  0  .  Then  f(x,y)  -  g(a^x  +  b^y) 
with  g(z)  -  f(j-,0)  .  One  way  to  show  this  is  to  fix  (x,  y)  and  define 


V  bi 

h(t)  -  f  [x  +  ~  y  -  ~  yt,  ty 

al  *1 


gCa^  +  b^) ; 


Then  h(0)  -  f| 

h(l)  -  f  (x,  y)  and  h’  (t)  =  0  ,  for  0  <.  t  <.  1 .  The  fundamental  theorem  of 
calculus  gives  h(l)  -  h’  +  h(0)  .  Suppose  the  result  is  true  for  operators 
of  degree  _<n  -  1  .  To  prove  it  for  degree  n  ,  write 

n-1 


^  K  3x  "  ai  3y  )  f  ”  {  (  bi  si  -  ai  3y)}  (  bn  L  “  an  By  )  f  E  0 


By  the  induction  hypothesis,  there  are  functions  g^,  1  <  i  <  n-1  satisfying 

n-1 


j,  «i<VV> 


A  solution  f  of  (2.1)  of  the  form 


n-1 


f*(x,  y)  -  [  h1(aix+  bty) 


i-1 


is  found  by  choosing  h^(t)  -  (bna^  -  a^^)”*-  g^sjds  •  This  is  well  defined 
because  the  lines  are  distinct.  Now  |  bQ  -  an  j  (f  -  f*>  -  0  can  be  solved 
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expllcltly  with  f  -  f  (x,  y)  ■  h  (a  x  +  b  y)  by  the  argument  for  n  ■  1  .  It 

n  n  n 

follows  that  f  ■  f  +  hQ  can  be  written  in  the  required  form.  ■ 

Remarks  on  Explicit  Computations.  Theorem  3  gives  the  existence  of  numbers 

3n 

cn,  ...,c  such  that  Ec.  — j - —  (f)  =  0  .  Fixing  n  +  1  distinct  pairs  (x. ,  y  ), 

v  n  I  r>.  in  n-i  i  1 


calculate 


3x±3ytl  1  j  {xA .  y±) 


3x3y 

and  solve  the  resulting  system  of  equations  for  c.  . 


It  is  feasible  to  check  if  the  polynomial  Cq+.-.  +  c^z  has  distinct  real  roots 

using  techniques  in  Chapter  6  of  Henrici  (1977).  Each  stage  of  the  procedure  is 

feasible  by  a  finite  algorithm.  If  the  procedure  fails  at  any  stage,  then 

equality  is  Impossible.  Given  feasible  crt,  ...,  c  ,  it  may  be  possible  to  find 

0  n 

the  roots  of  the  associated  polynomial.  This  determines  directions  (a^,  b^)  . 

In  simple  examples  there  is  often  enough  freedom  of  choice  to  make  deter¬ 
mination  of  (a^,  b^)  possible.  Consider  f(x,  y)  =  xy  for  n  *  2  , 


;  ,  3f  3f 

JI  b,  - - a.  — 

i-i  l  1  3x  1  3y 


V2  3,2  -  (bla2  +b2*l)  L\  +  V2  ■ 


Since  "  0  ,  =  1  ;  any  distinct  choice  of  a.  and  b  with 

3x^  Zy*  3xdy  1  i 

b^a^  =  ~^2al  wor'ts'  Taking  a^  *  b^  =  1,  a ^  *=  -b  =  1  ,  we  are  led  to  solve 


32f  T  32f 
3x2  3y2 


f(x»  y)  *  g^Cx  +  y)  +  g2(x-y) 


3  3  » 

Applying  -  -g—  to  both  sides  leads  to  y  -  x  *  2g2(x-  y)  ;  setting  y  =  0  ; 

1  X  x2  x^ 

g2(*)  “  ~~2*  *2  *  +  c2  •  Similarly,  g^(x)  ■  ~r  +  and  the  result  is 

12  12 

xy  *  ^(xfy)  +  c^  -  -^(x-y)  +  c2  where  c^  +  c2  *  0  is  forced.  In  general, 

lf  f  “  Ji  8i(aix+biy)  ;  h  -  ai  f  “  ci^a’1)(aix+biy)  »  for  811 

explicit  c^  .  This  determines  g^  up  to  an  essentially  free  choice  of  an 
n  -  1  degree  polynomial. 
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In  the  case  of  a  polynomial  f  ,  some  additional  tricks  become  available. 

b  n 

For  a  multinomial  xy  let  a  +  b  -  n  ;  only  sums  of  the  form  £  a  (x+0  y)“ 

1-1  1  J 

need  be  considered.  Expanding  out  and  equating  coefficients  gives 
Za^  -  0,  Za ^b^  -  0  ...  Zoij0®  -  -  ...  Za^b^  -  0  . 

( i) 

This  gives  n  +  1  equations  in  2n  unknowns.  These  are  linear  in  a  for  0^ 
given  and  may  be  solved  explicitly  because  the  matrix  is  a  Vandemonde  with  a 
well  known  inverse.  See  Goutschl  (1963). 

Proof  of  Theorem  4.  Condition  (1.4)  is  clearly  necessary.  For  sufficiency, 

observe  first  that  we  may  assume  that  the  normals  a^  to  the  planes  11^  span 

a  subspace  of  dimension  3  or  higher.  If  the  dimension  of  this  subspace  is 

2 

two,  then  the  problem  reduces  to  the  corresponding  problem  in  1R  which  was 
solved  la  Theorem  3.  The  proof  is  by  induction  on  n  .  For  n  *  1  ,  the 
argument  was  given  in  the  discussion  preceding  the  theorem.  Suppose  that  the 
result  has  been  demonstrated  for  n  -  1  .  For  i  -  1,  ...,  n,  let  b^  and 
fora  a  basis  for  11^  .  A  generic  element  of  can  be  written  as 
0jb^  +  ,  for  0^,  Y^  €  R  .  Write  the  equation  (1.4)  as 

(2.2)  (  J  Pi](Pi(f))  =  0  . 

By  the  Induction  hypothesis, 

n 

(2.3)  P.f  -  l  g . (a.  •  x)  . 

1  i-2  1  1 

We  will  now  find  the  general  solution  to  (2.3).  To  begin,  note  that  P^  may 
be  regarded  as  a  2-parameter  family  of  differential  operators  depending 
linearly  on  (fcj/Yj)  •  ft  follows  that  the  right  side  of  (2.3)  must  depend 
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linearly  on  .  Write 


and 


8i  "  6l8il  +  Yl8i2  * 


Plf  "  61 1  8il(ai *  x*  +  Yi 1  gi2(ai  *  x) 


For  this  equation  to  have  a  solution,  a  necessary  integrability  condition 
on  and  g^  must  be  satisfied.  To  see  this,  write 

P1  “  0l(  bll  3x^+  b12  b13  3^  +  Yl(  C11  3^+  c12  3x^+  c13  3xjJ  * 


From  (2.3)  it  must  be  that  the  following  two  equations  are  satisfied: 


l  bu  =  l  *ii(ai  * x) 

j»l  dXj  i_2  11 


t  cu'M~  m  !  &i2(ai  •  x)  • 

j-1  dxj  i-2  iZ  1 


The  necessary  condition  for  integrability  is 


:cIj  •*»  -  %  3^-t  E%2<*r  *> 


or 


<2'4)  j2  (C1  •  *l'8ll(*l  ’  *'  *  j2  (bi  ’  ‘  *>  ' 


Let  be  any  function  of  one  variable  such  that  Gi  "  8n  •  Then 


(2.5) 


'  1-2  611>1  '  ‘ll8il<a‘  '  X)  +  YU1  ‘  *l,8ll(*i  •  X) 
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Integrating  (2.4)  gives,  for  some  constant  k: 

J2  C1  '  *i  *U(V  x>  '  Jj  bl  •  *1  *i2(V  x>  +  k  ' 

Substituting  this  in  <2. 5)  gives 
n 

l  bl  *  ai  L68il(ai  *  x)  +  Y8i2(ai  *  x)l  +  Yk  • 

n  -1 

If  k  -  0,  then  a  particular  solution  to  P.f  •  Eg.  would  be  f  ■  J  (b.  •  a  )  G, 

1  1  i-2 

Note  we  can  assume  that  is  chosen  such  that  b^  •  a^  ^  0  for  all  i  *  2,  ....  n 
The  next  job  is  to  show  how  to  modify  G^  to  take  care  of  non-zero  k .  Let 

Pi<V  *2*  x35  ■  Gi^ai  *  x)  +  e±  *  * 
where  <5^  and  e^  *  (e^,  ei3^  be  cbosen  later.  Then, 

{  Z0blj  3x^  +  YClj  3^}tZFi(xl»  x2*  *3^ 

n  n 

-  I  (  bXj  •  atj)  C&g11(a1  •  x)  +  *  *>]  +  Yk  +  &±  { abj.  •  e±  +  •  e±)  . 


Set  6^-0  for  i  >.  3  .  Choose  such  that  e2  *  bi  “  G  and  e^  •  0  . 

Set  &2  *  -k/ECye2j  .  Then,  the  function 


f(x.,  x  ,  X- )  -  l 
i-2 


b  •  a 
i  i 


V*i 


x)  + 


V2 


n 

solves  the  equation  P.f  -  J  Bg..  +  Y8.,  •  Having  obtained  a  particular 

1  i-2 

solution,  we  proceed  to  describe  a  general  solution  of  P^f  -  g  by  studying 
the  general  solution  of  the  homogeneous  equation  P^f  -  0 .  But,  it  is  straight¬ 
forward  that  a  general  solution  of  the  homogeneous  equation  is  of  the  form 
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g(a^  •  x)  where  -  ai  •  ■  0  .  Thus  the  general  solution  of  (2.3)  is 

n  . 

(2.6)  l  r— 7—  G  (a  •  x)  +  5  e  *x  +  g(a  •  x)  . 

i-2  bi  *i  1  1  22  1 

In  (2.6)  e2  was  chosen  so  e2  •  b^  ■  0  f  .  Now  we  use  the  hypothesis 

that  three  of  the  vecotrs  a^  ,  say  a^,  are  linearly  independent,  and 

express  e^  as  a  linear  combination  of  these  three  vectors.  It  follows  that 
the  general  solution  can  be  expressed  in  the  form 

n 

l  H, (a.  •  x)  as  required.  ■ 
i-1 
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